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Logical meanings in multimedia learning 

MATERIALS: A MULTIMODAL DISCOURSE ANALYSIS 

George Vorvilas 


Abstract: Multimedia educational applications convey meanings through several semiotic modes 
(e.g. text, image, sound, etc.). There is an urgent need for multimedia designers as well as for 
teachers to understand the meaning potential of these artifacts and discern the communicative 
purposes they serve. Towards this direction, a hermeneutic semiotic framework is proposed, which 
provides a controlled vocabulary for describing the logical meanings among the components that 
constitute multimedia representations. The framework in turn is implemented in the analysis of 
some multimedia materials in order to detect the types of their components and the logico- 
semantic relations identified among them. Using that framework, teachers would be equipped with 
visual literacy skills for reading the logical meanings of multimedia learning materials and 
multimedia designers would be able to design their own applications with respect to the logical 
meanings they want to promote through them. 

Key words: Multimodal Discourse Analysis, Genres, Multimedia Educational Aplications, Logical 
meanings 

1. Introduction 

With the term “multimedia learning materials” we mean digital educational artifacts that represent and 
organize information through the combination of several semiotic resources (e.g. text, sound, image, 
etc.). These artifacts can be tutorials, drills and practices, simulations, educational games, etc. The 
modern teacher must be equipped with visual literacy skills and knowledge for being capable of 
interpreting the several multimodal meanings conveyed by these digital materials. What is more, 
multimedia designers must be also equipped with visual literacy skills for promoting multimodal 
meanings that make their learning materials more coherent and effective. Multimodal Discourse 
Analysis (MDA) can be a helpful tool towards this direction. For MDA, people use particular 
meanings to communicate with each other in specific social contexts. These meanings are created 
through complex \|/opPivaiiovq ocp visual, verbal aural, gestural, three-dimensional and other semiotic 
resources (O’ Halloran, 2008). MDA examines the ways several multimodal resources are integrated 
and interact with each other in specific social contexts (e.g. classrooms, online environments, etc.), in 
order to trigger several communicative functions (e.g. Unsworth, 2006; Jaipal, 2010; Karalis and 
Vorvilas, 2011). As regards multimedia learning material, semiotic resources can realize four kinds of 
meanings (O’ Halloran, 2008): 

• Experiential meaning , which concerns the ways the human experience of the world is visually 
or verbally represented in a multimedia message. 

• Logical meaning , which concerns the informational linking between multimedia components. 

• Interpersonal meaning , which concerns learners’ engagement and interaction with multimedia 
representations. 

• Textual meaning , which concerns the ways multimedia components are spatially and 
temporarily co-deployed on the multimedia representations’ layout. 
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In the present paper, through an analysis of several multimedia representations’ examples (MR), we 
focus on the ways logical meanings are organized, particularly on the logico-semantic relations that 
exist among the several semantic components that constitute a multimedia representation. 


2. Content aggregation of multimedia representations 

But what kind of semantic components constitute multimedia artifacts? Vorvilas, Vergidis and 
Ravanis (2011) have been proposed a hermeneutic semiotic framework according to which the digital 
content of multimedia artifacts can be distinguished to two main semantic units: items and clusters. 
Items are phonic, musical, visual and linguistic components such as push buttons, submit buttons, 
radio buttons, checkboxes, sliders, table cells, menu items, headings, titles, phrases, sentences, icons, 
symbols, sounds, etc. Items can be interpreted as communicative acts that offer, ask or demand 
information or goods and services. In figure 1 (MR1, 2013) we have marked in dashed circles and 
ellipses three such items. Item 1 is a switch that the user can turn on/off (service offering). Item 2 is a 
phrase that offers information (Current (I)). Item 3 is a question that demands information. 


Resistors 


Series and parallel \ Voltage divider \ Light dimmer \ 



Figure 1 . Items in a multimedia learning material 

Items can stand alone or they can be combined with each other to create clusters. The notion of cluster 
is used here to define local groupings of components with particular communicative functions on the 
multimedia layout (Baldry and Thibault, 2006). Clusters extend from smaller-scale groupings to 
larger-scale groupings, thus we could speak of micro-clusters and macro-clusters with respect to the 
level of the whole / part relations we want to describe among groupings. The communicative functions 
of clusters often allow us to classify them in particular genres. Genres are, generally speaking, types of 
texts or images that serve particular communicative goals (Table 1). 


Table 1 . Some genres and their communicative goals 


Genre types* 

Communicative goal 

Analytical representations 

To represent several entities in whole / part relations. 

Class ificational representations 

To represent several entities or phenomena through class / 
sub-class or co-class relations. 

Narrative representations 

To represent several actions, processes and changes. 

Procedures 

To tell someone how to do something. 

Reports 

To classify, describe or decompose several types of entities 
and phenomena. 

Explanations 

To explain why something happens. 

Stories 

To narrate, record or explain events and circumstances of the 
human life. 
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Questions-and-responses To demand information from someone and provide him with 

_the relevant feedback._ 

*Based on Kress and van Leeuwen, 2006, Martin and Rose, 2008, Allessi and Trollip, 2001 

Speaking more technically, according to Baldry & Thibault (2006) a cluster can be interpreted 
as an instantiation of a genre type. In this respect, the notion of cluster help us recognize 
several groupings as exemplary cases of genre types, but it can also help us handle hybrid 
cases in which a grouping although it serves a communicative goal it remains unclassified as 
a genre type. In such cases a grouping that does not instantiates any well-known genre type 
can be simply called “cluster”, without being attributed to any particular genre class. By well- 
known genres we mean here groupings that belong to a socially recognized genre classifying 
schema. From another perspective, we could say that clusters that are not classified under any 
particular genre class, they actually instantiate unclassified or unclassifiable genres (Santini, 
2007). 

Figure 2 depicts a cluster constituted of three sub-clusters marked in dashed rectangles (MR1, 
2013). Sub-cluster a consists of items such as one question, five statements, three circles that 
stand for radio buttons and a “Check” submit button. It instantiates the “alternate response” 
genre (a sub-category of the “questions-and-responses” genre - see Allesi and Trollip, 2001). 
Sub-cluster b consists of images linked with lines, words and numbers and the submit buttons 
named “Circuit symbols” and “View Symbols”. It instantiates the “analytical representations” 
genre. Sub-cluster c consists of a table with words and numbers accompanied with letters. It 
instantiates the “classificational representations” genre. The three sub-clusters together 
constitute a larger cluster in which they “co-operate” to help the user to extract information 
and submit his answer. That cluster cannot be classified with respect to the genre types 
presented in table 1 despite the fact that it serves a particular communicative function (it 
could also be considered that it might instantiate an unclassified or unclassifiable genre). 


Resistors Sortn <nd p«r«li<H Vo I tage divider Light dimmer 



Figure 2. Clusters in a multimedia learning material 


3. Logical meanings between multimedia components 

The meaningful information linking among components (items, clusters / sub-clusters) 
mentioned in the previous section, can be achieved through the logico-semantic relations of 
elaboration, extension, enhancement and projection, which can be traced through the 
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deployment of multimodal texts in general (e.g., Lemke, 2002; van Leeuwen, 2005; Djonov, 
2005; Kong, 2006; Martin and Rose, 2008; Vorvilas, Karalis and Ravanis, 2010; Vorvilas, 
Vergidis and Ravanis, 2011). 


logico-semantic relations 



Figure 3. A system of the logico-semantic relations between multimedia components 


3.1 Elaboration 

In elaboration one component elaborates the meaning of another, by describing it in detail, 
exemplifying it, clarifying it or restating it. Sub-categories of elaboration are: identification , 
explanation, exemplification and specification (Figure 3). 

Many times within a multimedia representation, readers must recognize several entities. Kong (2006) 
has used the term “identification” to underline the function of naming within a multimodal document. 

In explanation, one component explains the meaning of another through a different semiotic mode. 
Kong (2006) and van Leeuwen (2005) have used this term to describe cases in which one component 
restates the meaning of another. Martinec and Salway (2005) and Unsworth (2007) have used the term 
“exposition” to describe the same logico-semantic relation. 

Kong (2006), Martinec and Salway (2005) and Unsworth (2007) refer to the term “exemplification” to 
describe cases in which textual and pictorial components exemplify each other. In such cases the 
related components differ in generality, e.g. the textual component might be more general and the 
visual component might be more specific. 

Djonov (2005), Kong (2006) and van Leeuwen (2005) have used the term “specification” to underline 
the fact that one component can elaborate in detail the meaning of another. Djonov (2005) has pointed 
out that in specification the message of a component is elaborated by another through a shift in the 
degree of abstraction, given the fact that a transition from a more general to more specific information 
takes place. 

A variation of specification is the “overview / detail” relation (van Leeuwen, 2005), also described as 
“reinforcement” (Djonov, 2005). The “overview/detail” relation refers to the visual enlargement of 
several components within a multimodal representation. 
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3.2 Extension 

In extension, one component extends the meaning of another by adding new information, giving an 
exception to it or offering an alternative. Sub-categories of extension are: addition , variation and 
alternation. 

In addition, one component expands the meaning of another by adding new information which 
functions either in a complementary or a contrasting way. van Leeuwen (2005) has used the term 
“similarity” to describe the relation among filmic components when they share a similar information 
with respect to the content they represent. He has also used the term “contrast” to describe the 
relations among components when their meanings are in antithesis or contradiction. Djonov (2005) has 
also used the same terms to describe logico-semantic relations between hypertext components. 
Unsworth (2007) has made a distinction between “augmentation” and “divergence”, which we adopt 
here to describe logical meanings of addition. In augmentation, one component expands the meaning 
of another by adding new information. Divergence refers to cases in which several components deploy 
in parallel dissimilar and sometimes contradictory messages. 

Kong (2006) has used the term “variation” to describe relations of comparison / contrast among the 
messages of several components. Also, Kong (2006) has used the term “alternation” to describe cases 
in which one component replaces the meaning of another by providing alternative information that 
does not thoroughly changes the meaning of the original component. 

3.3 Enhancement 

In enhancement one component expands the meaning of another by enriching it with new information 
through circumstantial features of place, time, means, cause and condition. 

Djonov (2005), Martinec and Salway (2005) and Unsworth (2007) have used the term “place” to 
describe enhancement through information about time and Kong (2006) uses the term “spatio- 
temporal enhancement”. 

In order to describe temporal / logical sequencing among components, van Leeuwen (2005) has used 
the terms: “previous event”, “next event” and “simultaneous event”. 

In the logico semantic relation of means one component functions as a means for doing something 
related to another component. Kong (2006) has used the terms “manner” and “means” as sub- 
categories of enhancement in general and Djonov (2005) has classified “means” as sub-category of the 
“cause” relation. Unsworth (2007) has used the term “manner” as sub-category of the “means” 
relation. 

Among multimedia components we distinguish three cases of cause relations: effect, purpose and 
reason. In logico-semantic relations of effect (Kong, 2006), one component depicts, describes or 
shows the effect of an action (cause) described, depicted or triggered by another component. 

In the logico-semantic relation of purpose one component expresses an activity to be initiated in order 
to realize an intended situation with respect to another component (RST, 2014). The logico-semantic 
relation of purpose has been used by Kong (2006). Martinec and Salway (2005) have also regarded 
purpose and reason as sub-categories of causal enhancement. 

In logic-semantic relations of reason, a component explains why something described or depicted by 
another component happens. Djonov (2005) and Martinec and Salway (2005) have classified this 
relation as sub-category of “cause”. Djonov (2005) has reported as examples of reason relations in 
hypertext, cases in which the user takes feedback that explains why a specific choice he made was 
right or wrong. 

In the logico-semantic relation of condition, one component functions as a conditioning situation with 
respect to another component whose occurrence results from the occurrence of the conditioning 
situation (RST, 2014). This relation is mentioned in Unsworth (2007) for describing image-text 
relations. 
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3.4 Projection 

In projection the meaning of a component appears through another component either as idea or 
locution. When the second component of relation represents thoughts, projection is mental. When it 
represents speech, projection is verbal. 

4. Implementing the hermeneutic framework 

In this section, the semiotic framework described above is implemented in the analysis of several 
multimedia applications in order to detect the logical meanings among their components. Our analysis 
was based on examples from Wise-Online and Flexible Learning Toolboxes repositories which provide 
a variety of open multimedia learning resources. 

4.1. Identification 

In figure 4, we can see an example of identification relation among components. The list of items 
(labels) depicted on the right side of the screen identify the organelles’ names of the animal cell 
(analytical representation), each time the user rolls the cursor over several parts of the animated image 
(MR2, 2013). 



Mtochondnon ft frxXto witmbr t ot rtmciuw TT— no«r mmrr& T m n 

tofctod tf*0 propeftoro caiiod emu# Th* a «4t o t ATP tyr<r+%*% and a co*td iH 

Back *** PowerttouM of tbm ctfl \oxt 


Figure 4. An example of identification 


4.2. Explanation 

In figure 5, we can see an example of explanation: sub-cluster a (an instance of the “explanations” 
genre) explains the meaning of sub-cluster b (an instance of the “narrative representations” genre) by 
restating what exactly happens when the user increases pressure inside the piston by dragging the 
slider (MR3, 2013). Explanation relations can also be detected in figures 14 and 16, where the textual 
components (short sentences) explain the animated images. 
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BOYLE’S LAW 




This illustrates why the pressure varies as the volume changes, - .- . 

a .--'14.7 

If the gas is compressed, the molecules crowd closer 
together, which causes more molecular collisions with 
the container walls, resulting in an increase of pressure.! 


r HI 



, Back 


I atm 2 3 4 5 atm 

Slide the green Ixir to see how the volume affects pressure. 


Next 


M 


Figure 5. An example of explanation 


4.3. Exemplification 

In figure 6, the textual component a (an instance of the “reports” genre) describes the calyx and corolla 
of flowers. The pictorial component b (an instance of the “analytical representations” genre) functions 
as an example of the general class “flower” to which the textual component refers (MR4, 2013). 



A flower's petals compose the next whorl, 
the corolla, which usually has pigment 
Often the colorful petals attract pollinators. 
The calyx and corolla have no sexual function 


so are accessory parts of the flower 


Figure 6. An example of exemplification 


4.4. Specification 

In figure 7, we can see an example of specification. In the open window in a tutorial’s environment, 
the user can click on each one of the components (circuit symbols) depicted on the left side of the 
window so as to take detailed information on the right side. For example, when the user clicks on 
component a, the new sub-cluster b appears on the screen (an instance of the “reports” genre) 
providing him with detailed information about the meaning of the word “Transistor”. (MR1, 2013). 
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Figure 7. An example of specification 

An example of specification through the overview / detail device can be seen in figure 8: image b 
(analytical representation) provides a detailed elaboration of image a (analytical representation) by 
zooming in a part of it (MR5, 2013). 

Step 4: Measure the line side of the fuses that protect the 
Motor Control Transformer - single phase 480 volts. 

(The fuses ore located on the fuse bloc* in the middle of the 
cab«net on the n$ht side. The black wires are attached’) 


Line Side 



Load Side 


Figure 8. An example of specification (overview / detail) 


4.5. Addition 

In figure 9 we can see an example of augmentation. The depicted cluster consists of two sub-clusters: 
sub-cluster a (an instance of the “reports” genre) and sub-cluster b (an instance of the “narrative 
representations” genre). Both sub-clusters augment each other’s meaning since each one carries 
information not mentioned to the other. For example, sub-cluster b depicts information about the 
sexual lifecycle of plants not mentioned in sub-cluster a. Sub-cluster a also refers to information 
largely not depicted in sub-cluster b (MR6, 2013). 
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The Sexual Lifecycle of Plants 



Plant sexual cycles are more 
complicated than animal 
lifecycles because the plants 
exist as both haploid and 
diploid plants. Their lifecycle 
depends on an alternation 
of generation of haploid 
and diploid plants. 

a 

Plants also produce gametes 
(haploid cells)that come 
together during fertilization 
to restore the diploid state. 



Figure 9. An example of addition (augmentation) 

An example of divergence can be seen in figure 10: sub-cluster a (an instantiation of the “reports” 
genre) describes several aspects of plants and sub-cluster b (an instance of the “analytical 
representations” genre) depicts a shot of forest vegetation. Both sub-clusters deploy in parallel 
messages that differ in content (MR7, 2013). 


In order to support an increase m size, plants evolved complex polymers such as !ignm to 
strengthen cell walls, thus supporting heavier and larger structures. Plants evolved xytem tissue 
to carry water and phloem tissue to carry nutnents to these parts ~ 



Roots most likely evolved with mycorrhizae to anchor the plant and to absorb water and minerals 


Figure 10. An example of addition (divergence) 


4.6. Variation 

In figure 11, image b identified by the “Side View” item functions as a variation of image a identified 
by the “Top View” item. The user can compare / contrast two views of the same switch and its parts. 
Another example of variation can be seen in figure 14 between sub-clusters a and b, where two 
voltmeters move simultaneously, although in different speed, towards two magnetic fields. The two 
animated images promote their comparison / contrast since the user can observe that the faster the 
speed of the wires when they move through the flux the more the voltage increases and vice versa 
(MR5, 2013). 


Volume 7 Number 2, 2014 











34 


George Vorvilas 


Step 3: Proceed with testing the control circuit by starting with wire 1 of 
your line diagram, and continue to measure voltage drops across 
each switch and contact until you have examined each terminal 
attachment of every wire of your ladder diagram. (Remember to put 
one lead on the grounded conductor to read a voltage drop.) 



Side View 


START Top View 


The "Start" button 
is wired "Normally 
Open" (N.O.). 


Figure 11. An example of variation 


4.7. Alternation 

An example of alternation can be seen in figure 12, through the alternate “View symbols” and “View 
components” items (marked in dashed ellipses). Two different sub-clusters (analytical representations) 
of the same circuit can be depicted when the user clicks on these buttons: a “hybrid representation” 
(figure 12a), where the depicted items are close to the photorealistic code of picturing, and a 
“symbolic representation” (figure 12b), where the items are depicted through symbols that are used for 
representing them in the field of Physics (see for details: Dimopoulos, Koulaidis and Sklaveniti, 
2003). The two representations are alternative ways to represent an electric circuit (MR1, 2013). The 
same relation can be also detected between components a and c in figure 7. 



a 


b 


Figure 12. An example of alternation 


4.8. Place 

An example of spatial enhancement between components can be seen in figure 13. A map is depicted 
(an instance of the “analytical representations” genre), in which a storm is identified by the “Tropical 
Storm Erin” item (MR8, 2013). The storm is located in the Atlantic Ocean near the Turks & Caicos 
Islands and on the above right part of the representation a blue box with a list of items offers (among 
other things) information relevant to the location the storm evolves (“Latitude: 22.3 North”, 
“Longitude: 73.2 West”). 
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Current Conditions in Pensacola 


latiCpt 

Alabama 


Gulf of Mexico 





Joff XO, BOO PUSS) J 
Storm Locstron 
Ittude 223 North 
Longitude 2X2 We*t 
Preeeure 1004 mb 
Mod Speed 934 ktruhr 

Atlantic Ocean 


Tropical 
Storm 
/ Erin 

Ti fct&C*co« 


Figure 13. An example of spatial enhancement 


4.9. Time 

Examples of previous and next event between multimedia components can be seen in figures 1, 
4 and 5 where the “back” and “next” buttons are the items that allow the user to go forth and 
back between screens. In figure 14 sub-clusters a and b (instances of the “narrative 
representations” genre) are linked to each other through a “simultaneous event” relation 
because by clicking on the “Move Wire Down” button, the user starts an animation in which 
two voltmeters move simultaneously (although in different speed) towards two magnetic 
fields (MR9, 2013). 


Electrical Generator Basics 

Speed at which Conductor Cuts Flux Lines Page S of 11 



4.10. Means 

In figure 15 the user can drag item a (a slider) so as to increase or decrease the pressure inside the 
piston (sub-cluster b). This is an example of means relation between the instrument (slider) and sub¬ 
cluster b (MR3, 2013). 
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BOYLE'S LAW 

This illustrates why the pressure varies as the volume changes, fp 

If the gas is compressed, the molecules crowd closer 
together, which causes more molecular collisions with 
the container walls, resulting in an increase of pressure. 


Hr® 


• • 





! a 


I aim 2 3 4 $ atm 

Slide the green Ixir to see how the volume affects pressure. 

^\Back 


Next 


M 


Figure 15. An example of a means relation 


4.11. Cause 

An example of effect relation can be seen in figure 16: the user, by clicking the “Show Contact” 
button, sets in motion a moving object (item a) which touches and moves a lever. The result is the 
activation of cluster b (an electric circuit which instantiates the “analytical representations” genre) 
(MR 10, 2013). 



Page 3 of 16 


Page 3 of 16 


When the moving object touches and moves the lover, a sot of 
etectncal contacts ts forced either open or closed 


When the moving object touches and moves the lever, a set of 
etectncal contacts is forced either open or dosed 


slext 


Back 


Figure 16. An example of a cause-and-effect relation 

In figure 2 we referred to sub-cluster a (a “questions and responses” genre). When the user there clicks 
on the “submit” button he receives the answer depicted here, in figure 17. A small window informs 
him that his answer was correct (item a). A short text (sub-cluster b: an instance the “explanations” 
genre) explains the reason that item a holds true (MR1, 2013). 
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ffhat's correct, i ^ 


As the resistance 
increased, it made it 
harder for charged 
particles to flow through 
the circuit. This made the 
current decrease from 12 
amps to 6 amps. K 


0 « ) 




Figure 17. An example of a cause relation (reason) 

Figure 18 (detail of figure 4) gives us an example of purpose relation. Item a express an activity to be 
initiated in order to realize the intended situation of reading the organelles names that correspond to 
parts of the animated image of the cell. (MR2, 2013). 



Figure 18. An example of a cause relation (purpose) 


4.12. Condition 

An example of condition can be seen in figure 19. The user must give an answer to the “alternate 
response” genre (cluster a), that is, he must click on either the “True” or the “False” checkbox 
(conditioning situation) in order to go to the next screen, by clicking on the arrow button, otherwise he 
cannot continue because the arrow button is deactivated (MR11, 2013). The activation of the arrow 
button results from the occurrence of that conditioning situation. 
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4.13. Projection 

Examples of projection can be seen in figures 20a and b. In figure 20a, the information included in the 
speech balloon functions as a verbal projection of the words spoken to the user by the male figure 
depicted on the left (MR12, 2013). In figure 20b the cloud functions as a mental projection of the 
thoughts made by the female figure depicted on the left (MR13, 2013). 


^ ft you ftrai rw< 


9 

One# yowV* (onpMM ttn aUMTy or Mo rm go to If* Tm» 

s~r - Q 1 Can 1 change 1 


a b 

Figure 20. Examples of verbal and mental projection 


5. Conclusions 

What we proposed in the present paper was a general hermeneutic framework based on main logico- 
semantic relations that promote the logical linking among the components of multimedia learning 
materials. The analysis of the aforementioned examples was indicative, given the fact that more 
relations can be detected between the components of the selected multimedia representations. What is 
more, the types of relations explained here are not exclusive. On the contrary several other types have 
been proposed by many researchers for the description of logical meanings (e.g. Guijarro and Sanz 
(2009); Liu and O’Halloran, 2009; Zhao, 2010; Chan and Unsworth, 2011; Vorvilas, Karalis and 
Ravanis, 2011). The aim of the aforementioned analysis was to review through examples a basic 
vocabulary that would enhance teachers’ skills and knowledge in visual literacy, by making them 
aware of the meaning potential of multimedia learning materials and capable of identifying 
multimedia artifacts appropriate for their educational purposes.What is more, multimedia designers 
could be equipped with such a vocabulary in order to create and test learning materials (e.g. Vorvilas, 
2012; Vorvilas, 2013) following basic principles of multimodal discourse analysis. 
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